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Background/context : 

Description of prior research, its intellectual context and its policy context. 

The Stock Market Game™ is an educational program supported by the Securities 
Industry and Financial Markets Association (SIFMA) Foundation for Investor Education. The 
program is designed to teach students the importance of saving and investing by building their 
financial literacy skills. It is currently in use in all 50 states in the United States as well as in 
countries around the world and has been played by more than 1 1.5 million students. Beyond 
providing a unique educational opportunity, The Stock Market Game addresses this lack of 
financial literacy among today’s young adults and may provide additional benefits related to core 
academics (i.e., mathematics) and other educationally relevant skills such as better 
communication and conflict resolution. 

Research on financial literacy consistently demonstrates the widespread lack of student 
knowledge about financial matters (JumpStart Coalition, 2008), the demand for such courses 
from teachers and students (Charles Schwab Corporation, 2007), and the limited opportunities 
for students to take financial courses to expand their knowledge (Networks Financial Institute at 
Indiana State University, 2007). Furthermore, three states currently require one semester of 
instruction in financial literacy and 18 states require that instruction in personal finance be 
incorporated into other course matter (JumpStart Coalition, 2006). 

In mathematics, where achievement levels in the U.S. are low, little has changed over the 
past few decades. More than 25 years ago, the report, “A Nation at Risk” recommended at least 
three years of [high school] mathematics education — a requirement today in place in only 22 
states and the district of Columbia (U.S. Department of Education, 2006). Recent reports show 
that the United States places below average on country rankings of mathematics performance of 
high school students (Lemke, Sen, Pahlke, Partelow, Miller, Williams, 2004). And while 
elementary and middle school students tend to perform better, they nevertheless lag behind their 
international peers (Ginsberg, Cooke, Leinwand, Noell, & Pollock, 2005). 

Recently, Learning Point Associates conducted a nationwide randomized controlled trial 
assessing the impact of playing The Stock Market Game on student achievement in mathematics 
and investor knowledge. The study also collected data on students’ and teachers’ perception of 
the benefits of the program. 

Purpose / objective / research question / focus of study: 

Description of what the research focused on and why. 

The primary focus of the study was to measure the impact of playing the Stock Market 
Game on student achievement in investor knowledge and mathematics. Since it is expected that a 
supplemental education program will improve students’ understanding of program-related 
content, we expected that students who played The Stock Market Game would improve their 
knowledge of topics related to savings and investing. A review of program content, however, 
revealed the mathematical nature of The Stock Market Game, suggesting another possible 
expectation related to student performance. Namely, is the mathematical component of the 
program material strong enough to affect students’ mathematics skills? We designed the study in 
order to answer two research questions about impact: 
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1. What is the impact of The Stock Market Game on student academic achievement in 
mathematics for students in Grades 4-10? 

2. What is the impact of The Stock Market Game on student investor knowledge for 
students in Grades 4-10? 

Setting: 

Description of where the research took place. 

This RCT began in the spring of 2008 with participant recruitment; the actual study trial 
occurred in the first academic semester of the 2008-09 school year. During the prior academic 
year (2007-08) instruments were developed, pilot tested and revised according to the results of 
psychometric analysis. Teachers from all around the country were invited to participate in the 
study, and the final sample of classrooms included treatment classrooms in 43 states and control 
classrooms in 47. Student test data were collected via online tests which were administered by 
teachers. 

Population / Participants / Subjects: 

Description of participants in the study: who (or what) how many, key features (or characteristics). 

Approximately 1,200 teachers signed up to participate in the study. Of these, 823 met the 
eligibility requirements (i.e., teaching a 10 or 15-week game and teaching students in Grades 4- 
10) and were randomly assigned to participate in the study as either treatment or control 
classrooms. Of these 823 assigned, 568 teachers confirmed participation (296 treatment and 272 
control). 

Of these 568 teachers, 555 teachers submitted student test data. There were 522 
classrooms that provided student test data for the investor knowledge test (269 treatment, 253 
control), and 509 classrooms that provided student test data for the mathematics tests (265 
treatment, 244 control). Table 1 summarizes recruitment and participation in this RCT. (Please 
insert table 1 here.) 

Treatment and control groups did not differ significantly on any of the demographic 
characteristics for which we had data (e.g., years of classroom teaching experience, years of 
experience teaching The Stock Market Game, locale, region of the country, percent minority, 
percent economically disadvantaged, etc.). Likewise, treatment and control classrooms for 
which assessment data were not received were not different in terms of any of these 
characteristics. 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 

The Stock Market Game centers on teams of students managing real-time virtual 
(Internet-based) investments. Students use research and program-provided news updates to 
invest a hypothetical $100,000 in a portfolio, which simulates the results of their investments as 
though it were in the real marketplace. The program offers several game sessions in the fall and 
spring of each school year, most of which are 10 to 15 weeks in duration, with some that run for 
the full academic year. During these sessions, students compete with teams in their classroom 
and teams in their states to increase the value of their portfolio. This competition adds to the 
appeal of the program for many students, and the team structure provides the opportunity to build 
interpersonal skills, such as leadership and negotiation. 
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Research Design: 

Description of research design (e.g., qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

We conducted a nationwide randomized controlled trial (RCT) assessing the impact of 
playing The Stock Market Game on student mathematics achievement and investor knowledge. 
Prior to the study, tests of academic content were developed, administered, psychometrically 
evaluated, and, where appropriate, revised. For the study, classrooms of students were randomly 
assigned to play or not to play the game and students in both treatment and control classrooms 
were assessed on their academic skill level before and after the game period. After 
implementation of the RCT, final data were analyzed psychometrically to create summarized 
scale scores and statistically using hierarchical linear modeling to measure program impact and 
implementation. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

Several age appropriate tests were created in order to measure students’ mathematics and 
investor knowledge. The mathematics tests were designed for students in grades 4-6 and 7-10 
and the investor knowledge tests were designed for students in grades 4-5, 6-8, and 7-10. 

Both the mathematics and investor knowledge tests included pretest and posttest versions 
which contained a number of overlapping items. The mathematics tests were constructed using 
publicly available items from the National Assessment of Educational Progress (NAEP) and 
assessed concepts related to the content of The Stock Market Game including: number properties 
and operations; measurement; geometry; data analysis and probability; and for the older students 
algebra as well. The investor knowledge tests were developed with the help of experts in the 
area of testing financial literacy, aligned to the curriculum content of The Stock Market Game, 
and measured understanding in: economic concepts; investment strategies; investor research; and 
calculations. 

The tests were administered online before and after The Stock Market Game fall session. 
Psychometric analyses using the Rasch model (Rasch, 1980; Wright & Masters, 1982) were 
conducted on a sample of student test and survey data collected prior to conducting the RCT. 
These analyses were used to examine the reliability and validity of the instruments and to 
facilitate revisions designed to increase their overall utility. After the data were collected during 
the RCT phase of the study, the final study instruments were again analyzed to examine the 
reliability and validity of the scores produced by the instruments. In addition, the pretests and 
posttests were equated to one another, and scale scores were developed for use in the statistical 
models of student outcomes. 

This study yielded results from five tests: two mathematics tests and three investor 
knowledge tests. The analytic methods used to examine the data from these five tests were the 
same. Scaled data were fit with multilevel models generating intent-to-treat and treatment-on- 
the-treated estimates. 

First, we analyzed the data by comparing the outcomes for the control and treatment 
groups, regardless of whether participants actually received the treatment generating the ITT 
estimates. In this case, the ITT estimates provided the likely impact on student achievement for 
students who are assigned to play The Stock Market Game. In reality, compliance with random 
assignment is never perfect. In this study there may have been some classes assigned to play the 
game that did not play, and some classes that were assigned not to play but actually did play. To 
estimate the impact of playing the game, while also accounting for treatment crossover, we used 
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an instrumental variables approach as outlined by Angrist, Imbens & Rubin, (1996). In this case 
the ToT estimates provided the likely impact on student achievement for students who were 
assigned to play and did play The Stock Market Game. 

For both the ITT and ToT estimates, we fit two mixed models: The first modeled 
achievement based only on the experimental indicator (i.e., treatment and control). Because 
analysis of the some of the pretests indicated differences between the treatment and control 
groups, a second model was fit to test for program impact adjusting for pretest abilities. All 
models were fit using the MIXED procedure in SAS 9.0. 

Findings / Results: 

Description of main findings with specific details. 

Playing The Stock Market Game had a positive impact on student learning in 
mathematics for students in grades 4-6. Although the impact estimates from each of the models 
varied in magnitude, all were statistically significant with the treatment group outperforming the 
control group. Table 2 shows the estimates in terms of scale score differences and their 
significance levels. These estimates range approximately from 23 to 41 scale-score points. 

The findings for the impact of playing The Stock Market Game on student learning in 
mathematics for students in Grades 7-10 were inconsistent across the various modeling efforts. 
However, the estimates were significant and positive in those models which included a prior 
measure of skill level. Table 2 shows the impact estimates in terms of scale score differences and 
their significance levels for each of the models. These estimates range approximately from 12 to 
15 scale-score points. (Please insert table 2 here.) 

Playing the Stock Market Game had a positive impact on student learning of concepts 
related to saving and investing for students in all grades. Students in the treatment groups 
outperformed students in the control groups. Although the impact estimates varied in magnitude, 
all were statistically significant. Table 3 shows the impact estimates in terms of scale score 
differences and their significance levels. These estimates range from approximately 53 points to 
62 points for students in grades 4-5, from 41 to 58 points for students in grades 6-8 and from 
approximately 29 to 32 points for students in grades 9-10. (Please insert table 3 here.) 

Conclusions: 

Description of conclusions and recommendations based on findings and overall study. 

In a recent paper examining effect-size change for a number of standardized tests, Hill, 
Bloom, Black, and Lipsey (2007), derive empirical average effect sizes for students in Grades 
K-12 over the course of a year in both mathematics and reading. Hill et al. further provide 
average effect sizes from the results of more than 60 randomized controlled trials that are 
categorized according to grade level and the specificity of the test. The empirical benchmarks 
from this recent report are used to interpret the magnitude, and meaning, of the effect size 
estimates for the impact of playing The Stock Market Game. 

Since the results were similar for all models we interpret the effect sizes of the ToT 
models that included a prior measure of student skill. We chose this model because when 
statistically examined there were performance differences on the pretest for several of the tests. 

The effect sizes estimates for the ToT models of mathematics were ES= 0.25 for students 
in grades 4-6 and ES=0 . 1 7 for students in grades 7-10. In the absence of an intervention, during 
the course of one full year, students in grades 4-6 might demonstrate an effect-size change in 
mathematics between 0.30 and 0.56 (Hill et al., 2007). Students in grades 7-10 might experience 
gains between 0.14 and 0.32. The effect sizes demonstrated by the RCT of 10 and 15 week 
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playing of The Stock Market Game fall within the range of average annual growth. It is 
important to note that the Hill et al. (2007) report based growth on nationally normed 
standardized tests and nationally derived standard deviations. The explanation of the effect size 
finding for the mathematics tests used in this study does not imply growth on a standardized test. 
Comparison of the study effect sizes to the Hill et al. report still provides one frame of reference 
for interpretation of these effect sizes. 

Another interpretation of these effect sizes comes from a comparison with other 
randomized studies. Hill et al. report the average effect sizes for randomized studies according to 
three types of tests for elementary students: standardized test (broad), standardized test (narrow), 
and specialized topic/test. We consider the mathematics test to fall into the category of 
standardized test (narrow). For students in elementary schools who are tested with a standardized 
test that has a narrow focus, the average effect size for the intervention is 0.23. Therefore, one 
might consider the impact of The Stock Market Game to have an average effect on mathematics 
achievement. For high school students the average effect size for randomized studies is 0.27, 
somewhat higher than the effect size we report for the impact of playing The Stock Market 
Game. This comparison suggests that the game has slightly less than an average impact on 
mathematics achievement for older students. 

The effect sizes for investor knowledge were ES=0 . 43 for students in grades 4-5; 

ES= 0.45 for students in grades 6-8 and £3=0.39 for students in grades 9-10. The average effect 
size of other randomized studies assessed with a specialized test for elementary school students 
is £3=0.44 (Hill et al. (2007). Therefore, playing The Stock Market Game could be considered to 
have an effect similar to that of other targeted interventions for students in grades 4-5. For 
interventions targeted at middle schools students the average effect size is £3=0.51. This average 
effect size is slightly higher than that demonstrated by the Stock Market Game study. For 
interventions targeted at high school students, the average effect size is 0.27, an estimate which 
is lower than that reported by our study. (Please insert table 4 here.) 

Taken together, the results of this study are promising. The findings suggest that this 
supplemental program designed to teach students about an important topic, one that many states 
are beginning to consider a requirement — saving and investing, can help students to leam, not 
only about topics within financial literacy, but also core aspects of mathematics. 
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Table 1 



Summary ofRCT Recruitment ancl Participation 



Group 


Recruited 




Participated 


Confirmed 


Mathematics 


Investor Knowledge 








265 


269 


Treatment 


406 


296 


65% of recruited 


66% of recruited 








90% of confirmed 


91% of confirmed 








244 


253 


Control 


417 


272 


59% of recruited 


61% of recruited 








90% of confirmed 


93% of confirmed 



Table 2 



Impact Estimates for Mathematics Learning 





Grades 4-6 


Grades 7- 


10 




n: 


=4,358 


n=4,500 




Models 


ITT 


ToT 


ITT 


ToT 


Treatment Indicator 


41.61*** 


37.21*** 


13.91 


15.26 


Treatment Indicator + Pretest 


23.16*** 


26.88*** 


12.33* 


14.52* 



* Significant a the p<.05 level. 
**Significant at the p<.01 level. 
***Significant at the p<.001 level. 



Table 3 



Impact Estimates for Investor Knowledge Learning 





Grades 4-5 
n=2,616 


Grades 6-8 
n=4,275 


Grades 9- 
n=1 ,703 


10 


Models 


ITT 


ToT 


ITT 


ToT 


ITT 


ToT 


Treatment Indicator 


55.23*** 


52.93*** 


57.74*** 


51.43*** 


34.37*** 


29.47* 


Treatment Indicator 
+ Pretest 


55.77*** 


61.51*** 


41.07*** 


42.75*** 


31.69*** 


36.12** 



Table 4 



Effect Size and Confidence Intervals of the ToT Models (with pretest ) 



Test 


Effect Size 


Confidence Interval 


Mathematics, Grades 4-6 


0.25 


0.11-0.40 


Mathematics, Grades 7-10 


0.17 


0.02-0.32 


Investor Knowledge, Grades 4-5 


0.43 


0.24-0.61 


Investor Knowledge, Grades 6-8 


0.45 


0.29-0.60 


Investor Knowledge, Grades 9-10 


0.39 


0.16-0.62 
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